Driver drowsiness monitoring using eye movement features derived from electrooculography

The increase in vehicle accidents due to the driver drowsiness over the last years highlights the need for developing reliable drowsiness assistant systems by a reference drowsiness measure. Therefore, the thesis at hand is aimed at classifying the driver vigilance state based on eye movements using electrooculography (eog). In order to give an insight into the states of driving, which lead to critical safety situations, first, driver drowsiness, distraction and different terminologies in this context are described. Afterwards, countermeasures as techniques for keeping a driver awake and consequently preventing car crashes are reviewed. Since countermeasures do not have a long-lasting effect on the driver vigilance, intelligent driver drowsiness detection systems are needed. In the recent past, such systems have been developed on the market, some of which are introduced in this study. As also stated in previous studies, driver state is quantifiable by objective and subjective measures. The objective measures monitor the driver either directly or indirectly. For indirect monitoring of the driver, one uses the driving performance measures such as the lane keeping behavior or steering wheel movements. On the contrary, direct monitoring mainly comprises the driver’s physiological measures such as the brain activities, heart rate and eye movements. In order to assess these objective measures, subjective measures such as self-rating scores are required. This study introduces these measures and discusses the concerns about their interpretation and reliability. The developed drowsiness assistant systems on the market are all based on driving performance measures. These measures presuppose that the vehicle is steered solely by the driver himself. As long as other assistance systems with the concept to keep the vehicle in the middle of the lane are activated, driving performance measures would make wrong decisions about warnings. The reason is what the sensors measure is a combination of the driver’s behavior and the activated assistance system. In fact, the drowsiness warning system cannot determine the contribution of the driver in the driving task. This underscores the need for the direct monitoring of the driver. Previous works have introduced the drop of the alpha spindle rate (asr) as a drowsiness indicator. This rate is a feature extracted out of the brain activity signals during the direct monitoring the driver. Additionally, asr was shown to be sensitive to driver distraction, especially a visual one with an counteracting effect. We develop an algorithm based on eye movements to reduce the negative effect of the driver visual distraction on the asr. This helps to partially improve the association of asr with the driver drowsiness. Since the focus of this study is on driver eye movements, we introduce the human visual system and describe the idea of what and where to define the visual attention. Further, the structure of the human eye and relevant types of eye movements during driving are defined. We also categorize eye movements into two groups of slow and fast eye movements. We show that blinks, in principle, can belong to both of these groups depending on the driver’s vigilance state. eog as a tool to measure the driver eye movements allows us to distinguish between drowsinessor distraction-related and driving situation dependent eye movements. Thus, in a pilot study, xvi Abstract an experiment under fully controlled conditions is carried out on a proving ground to investigate the relationship between driver eye movements and different real driving scenarios. In this experiment, unwanted head vibrations within eog signals and the sawtooth pattern (optokinetic nystagmus, okn) of eyes are realized as situation dependent eye movements. The former occurs due to ground excitation and the latter happens during small radius (50m) curve negotiation. The statistical investigation expresses a significant variation of eog due to unwanted head vibrations. Moreover, an analytical model is developed to explain the possible relationship of okn and tangent point of the curve. The developed model is validated against the real data on a high curvature track. In order to cover all relevant eye movement patterns during awake and drowsy driving, different experiments are conducted in this work including daytime and nighttime experiments under real road and simulated driving conditions. Based on the measured signals in the experiments, we study different eye movement detection approaches. We, first, investigate the conventional blink detection method based on the median filtering and show its drawback in detecting slow blinks and saccades. Afterwards, an adaptive detection approach is proposed based on the derivative of the eog signal to simultaneously detect not only the eye blinks, but also other driving-relevant eye movements such as saccades and microsleep events. Moreover, in spite of the fact that drowsiness influences eye movement patterns, the proposed algorithm distinguishes between the often confused driving-related saccades and decreased amplitude blinks of a drowsy driver. The evaluation of results shows that the presented detection algorithm outperforms the common method based on median filtering so that fast eye movements are detected correctly during both awake and drowsy phases. Further, we address the detection of slower eye blinks, which are referred to as typical patterns of the drowsiness, by applying the continuous wavelet transform to eog signals. In our proposed algorithm, by adjusting parameters of the wavelet transform, fast and slow blinks are detected simultaneously. However, this approach suffers from a larger false detection rate in comparison to the derivative-based method. As a result, for blink detection in this work, a combination of these two methods is applied. To improve the quality of the collected eog signals, the discrete wavelet transform is benefited to remove noise and drift. For the noise removal, an adaptive thresholding strategy within the discrete wavelet transform is proposed which avoids sacrificing noise removal for saving blink amplitude or vice versa. In previous research, driver eye blink features (blink frequency, duration, etc.) have shown to be correlated to some extent with drowsiness. Hence, within a level of uncertainty they can contribute to driver drowsiness warning systems. In order to improve such systems, we investigate characteristics of detected blinks with respect to their different origins. We observed that in a real road experiment, blinks occur both spontaneously or due to gaze shifts. Gaze shifts between fixed positions, which occurred due to secondary visuomotor task, induced and modulated the occurrence of blinks. Moreover, the direction of the gaze shifts affected the occurrence of such blinks. Based on the eye movements during another experiment in a driving simulator without a secondary task, we found that the amount of gaze shifts (between various positions) is positively correlated with the probability of the blink occurrence. Therefore, we recommend handling gaze shift-induced blinks (e.g. during visual distraction) differently from those occurring spontaneously in drowsiness warning systems that rely solely on the variation of blink frequency as a driver state indicator. After studying dependencies between blink occurrence and gaze shifts, we extract 19 features out of each detected blink event of 43 subjects collected under both simulated and real driving Abstract xvii conditions during 67 hours of both daytime and nighttime driving. This corresponds to the largest number of extracted eye blink features and the largest number of subjects among previous studies. We propose two approaches for aggregating features to improve their association with the slowly evolving drowsiness. In the first approach, we solely investigate parts of the collected data which are best correlated with the subjective self-rating score, i.e. Karolinska Sleepiness Scale. In the second approach, however, the entire data set with the maximum amount of information regarding driver drowsiness is scrutinized. For both approaches, the dependency between single features and drowsiness is studied statistically using correlation coefficients. The results show that the drowsiness dependency to features evolves to a larger extent non-linearly rather than linearly. Moreover, we show that for some features, different trends with respect to drowsiness are possible among different subjects. Consequently, we challenge warning systems which rely only on a single feature for their decision strategy and underscore that they are prone to high false alarm rates. In order to study whether a single feature is suitable for predicting safety-critical events, we study the overall variation of the features for all subjects shortly before the occurrence of the first unintentional lane departure and first unintentional microsleep in comparison to the beginning of the drive. Based on statistical tests, before the lane departure, most of the features change significantly. Therefore, we justify the role of blink features for the early driver drowsiness detection. However, this is not valid for the variation of features before the microsleep. We also focus on all 19 blink-based features together as one set. We assess the driver state by artificial neural network, support vector machine and k-nearest neighbors classifiers for both binary and multi-class cases. There, binary classifiers are trained both subject-independent and subject-dependent to address the generalization aspects of the results for unseen data. For the binary driver state prediction (awake vs. drowsy) using blink features, we have attained an average detection rate of 83% for each classifier separately. For 3-class classification (awake vs. medium vs. drowsy), however, the result was only 67%, possibly due to inaccurate self-rated vigilance states. Moreover, the issue of imbalanced data is addressed using classifier-dependent and classifier-independent approaches. We show that for reliable driver state classification, it is crucial to have events of both awake and drowsy phases in the data set in a balanced manner. The reason is that the proposed solutions in previous researches to deal with imbalanced data sets do not generalize the classifiers, but lead to their overfitting. The drawback of driving simulators in comparison to real driving is also discussed and to this end we perform a data reduction approach as a first remedy. As the second approach, we apply our trained classifiers to unseen drowsy data collected under real driving condition to investigate whether the drowsiness in driving simulators is representative of the drowsiness under real road conditions. With an average detection rate of about 68% for all classifiers, we conclude their similarity. Finally, we discuss feature dimension reduction approaches to determine the applicability of extracted features for in-vehicle warning systems. On this account, filter and wrapper approaches are introduced and compared with each other. Our comparison results show that wrapper approaches outperform the filter-based methods.xvii conditions during 67 hours of both daytime and nighttime driving. This corresponds to the largest number of extracted eye blink features and the largest number of subjects among previous studies. We propose two approaches for aggregating features to improve their association with the slowly evolving drowsiness. In the first approach, we solely investigate parts of the collected data which are best correlated with the subjective self-rating score, i.e. Karolinska Sleepiness Scale. In the second approach, however, the entire data set with the maximum amount of information regarding driver drowsiness is scrutinized. For both approaches, the dependency between single features and drowsiness is studied statistically using correlation coefficients. The results show that the drowsiness dependency to features evolves to a larger extent non-linearly rather than linearly. Moreover, we show that for some features, different trends with respect to drowsiness are possible among different subjects. Consequently, we challenge warning systems which rely only on a single feature for their decision strategy and underscore that they are prone to high false alarm rates. In order to study whether a single feature is suitable for predicting safety-critical events, we study the overall variation of the features for all subjects shortly before the occurrence of the first unintentional lane departure and first unintentional microsleep in comparison to the beginning of the drive. Based on statistical tests, before the lane departure, most of the features change significantly. Therefore, we justify the role of blink features for the early driver drowsiness detection. However, this is not valid for the variation of features before the microsleep. We also focus on all 19 blink-based features together as one set. We assess the driver state by artificial neural network, support vector machine and k-nearest neighbors classifiers for both binary and multi-class cases. There, binary classifiers are trained both subject-independent and subject-dependent to address the generalization aspects of the results for unseen data. For the binary driver state prediction (awake vs. drowsy) using blink features, we have attained an average detection rate of 83% for each classifier separately. For 3-class classification (awake vs. medium vs. drowsy), however, the result was only 67%, possibly due to inaccurate self-rated vigilance states. Moreover, the issue of imbalanced data is addressed using classifier-dependent and classifier-independent approaches. We show that for reliable driver state classification, it is crucial to have events of both awake and drowsy phases in the data set in a balanced manner. The reason is that the proposed solutions in previous researches to deal with imbalanced data sets do not generalize the classifiers, but lead to their overfitting. The drawback of driving simulators in comparison to real driving is also discussed and to this end we perform a data reduction approach as a first remedy. As the second approach, we apply our trained classifiers to unseen drowsy data collected under real driving condition to investigate whether the drowsiness in driving simulators is representative of the drowsiness under real road conditions. With an average detection rate of about 68% for all classifiers, we conclude their similarity. Finally, we discuss feature dimension reduction approaches to determine the applicability of extracted features for in-vehicle warning systems. On this account, filter and wrapper approaches are introduced and compared with each other. Our comparison results show that wrapper approaches outperform the filter-based methods.